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ABSTRACT 

Evaluation in continuing education must be 
continuous, not a one-shot process. Steps in evaluating include: 
preparing goals and objectives, selecting starting points appropriate 
to the clients, determining programs and attitudes, measuring 
progress, interpreting evidence of progress, and then using the 
evaluation to help students by providing guidance and motivation and 
helping teachers to clarify objectives and plan instruction. 
Evaluation programs are not used widely for several reasons: 1) It is 
still not perceived as a necessary part of the professional 
activities of program developers; 2 ) program developers regard it as 
of lower priority than their other activities; 3) it commands little 
financial support; 4) program developers often feel they are 
incompetent to carry out such evaluations; 5) evaluation is 
threatening to some program developers; 6) a truly professional 
education climate may not exist in the program development unit. 

(JK) 



o 



o if S-&8 ED 059590 



© 1969 by American Journal ol' Pharimu’i-ut ical 

Education , Vol. 33, No. 5. 

Reproduced by permission. 



"PERMISSION TO REPRODUCE THIS COPY- 
RIGHTED MATERIAL HAS BEEN GRANTED 
BY 

Chart's A • W t: A gvMgy 



TO ERIC ANO ORGANIZATIONS OPERATING 
UNDER AGREEMENTS WITH THE US OFFICE 
OF EDUCATION. FURTHER REPRODUCTION 
OUTSIDE THE ERIC SYSTEM REQUIRES PER- 
MISSION OFTHE COPYRIGHT OWNER *' 



U.S. DEPARTMENT OF HEALTH. 

EDUCATION & WELFARE 
OFFICE OF EDUCATION 
THIS OOCUMENT HAS BEEN REPRO- 
OUCEO EXACTLY AS RECEIVF.O FROM 
THE PERSON OR ORGANIZATION ORIG- 
INATING IT POINTS OF VIEW OR OPIN- 
IONS STATEO DO NOT NECESSARILY 
REPRESENT OFFICIAL OFFICE OF EDU- 
CATION POSITION OR POLICY. 



CHARLES A. WEDEMEYER* 



EVALUATION OF CONTINUING EDUCATION PROGRAMS 



1 here are personal needs and reasons 
for evaluating programs. We know that 
evaluation is an habitual and necessary 
activity of all of us. At one level, either 
consciously or unconsciously, our senses 
continually assess the state of being of 
our bodies: arc we hungry? tired? in 
balance? hot? cold? comfortable? tense? 
Each moment of our lives we measure 
our state of being against the ideal 
euphoria which we all seek. 

At another level, euphoria is more 
than physical well-being and involves 
assessing or appraising the activities of 
man measured against his aspirations. 
We are all familiar with the frequent 
readings we take to sec “where we arc” 
in our carccrs, in our community, in our 
families; in our friendship circles, etc. 

Wc are so accustomed to continual 
status probes of a pcrson.nl kind that wc 
carry over these behaviors to our pro- 



fessional activities. “How am I doing?” 
becomes “How is the Institute on Drug 
Abuse going?" or “How is the inde- 
pendent study course on pharmacology 
succeeding?" 

Our professional needs for evaluation 
are, of course, linked to our personal 
needs. On the one hand, this close per- 
sonal linkage is tnc source of the moti- 
vation necessary to carry out the status 
probe; but, cm the other hand, it is also 
the source of bias and internal conflict 
of interest which may delude us with 
illusions of success or confuse us in 
the open search for improvement. Wc 
want to succeed; we want to know how 
things are going; hut our very hopes for 
success may cause us to misread the 

* William //. URhty Professor of Education, 
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help ns assess what is going on. We 
may ask students to rale or react to what 
we are doing. These friend or consumer 
indices may confirm or not confirm what 
we have noticed ourselves. Whatever the 
feedback, we lend, wisely I think, to 
place more credence in these outside 
reactions than in our own judgments. 
Indeed, if we have not been able to 
mount a more objective evaluation of 
our work, we are usually quite willing 
to place a reliance on these “happiness 
indices” which may also be unwarranted 
and self-deluding. Friends, colleagues — 
even students — lend to give back to us 
what we want to hear. “F.vcn your best 
friend won't tell yon.” But when de- 
cisions must he made on the basis of 
what is actually li<ippcninf >— -the reality 
of success and failures in programs — we 
usually know that the informal judg- 
ments of friends and the happiness in- 
dices of consumers are not reliable eval- 
uative devices, even though we will con- 
tinue. wistfully, to listen to them. 

We turn to more formal and sophis- 
ticated evaluative processes because of 
needs which go beyond those personal 
and professional needs mentioned ear- 
lier. We have to make decisions, or our 
superiors have to make decisions, and 
we and they need objective information 
of greater validity and reliability. We 
need to be sure that we know what is 
actually happening in our programs. 
Are we achieving our objectives? How 
well? Where did things go awry? Why? 
What can we learn about causes for un- 
expected results? Should wc reschedule 
the same institute, with the same staff 
and same agenda, or revise it, or start 



anew in an effort to make it more 
relevant, more useful, more goal-di- 
rected? 

Th, •odore Roelhkc, the contemporary 
American poet who has said so many 
things so pungent ly. wrote two lines 
which lit into our discussion of evalua- 
tion. He said: “There’s nothing like ig- 
norance to engender wild enthusiasm.” 
and “lie who is willing to he vulnerable 
moves among mysteries.” Take the 
second one first: Continuing education 
requires us to be innovators — in sub- 
ject matters for adult pharmacists in a 
changing world, in format and method- 
ology in a period of fast-moving educa- 
tional technology. The innovator is vul- 
nerable. He is visible because his neck 
is way out. He moves among mysteries. 
The unknowns surround him because be 
is breaking new ground. 

Ignorance of what is actually happen- 
ing in programs can engender wild en- 
thusiasm when perhaps skeptical disen- 
chantment is what real data would indi- 
cate as a basis for decision making. 

Sound evaluation can give at least 
a partial sense of security to educator- 
innovators — even if results are negative, 
because the professional has confidence 
that he is indeed doing a professional 
job. and his colleagues and superiors 
know it also. After all. a mark of the 
professional is process as well as content 
orientation. Sound evaluations get into 
the communications channels and pro- 
vide an appropriate and restrained kind 
of visibility for the professional which 
encourages further program develop- 
ment. supplies new goals, energizes goal- 
oriented behavior, and enhances careers 

(I). 

We all want the gentle rain of subsidy 
to fall upon and nourish. our programs. 
In an older and simpler time, subsidy 
was in many ways easier to get and 
keep. But things arc not that way any 
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longer. There is a growing .scientism al 
all levels of American education, anil 
powerful forces are at work which have 
altered perhaps forever the simple secur- 
ities and controls which educators once 
had over their programs (2). This is 
an era of dig business in education; dig 
Government in education; dig Objectives 
in education; and dig Management in 
education. Management science and cost 
accounting have already made their ap- 
pearance. and continuing subsidy more 
and more is dependent upon a truly 
professional appraisal of what has been 
done, what has happened, the goals 
achieved, the yields of controlled exper- 
iments. and cost studies. The lack of re- 
liable evaluation can often, now. result 
in a drought of subsidy instead of that 
steady rain so fervently wished for. The 
situation today is reminiscent of that 
wonderful scene in Paradise Lost where 
the Devil Satan is seeking to arouse his 
fallen followers who lie abject in hellish 
misery. He cries to them, "Awake, arise, 
or be forever fallen (3)!” Satan’s 
words complete the analogy, for to many 
in academia, the promptings they now 
hear to engage in more scientific evalu- 
ation seem to have an origin at least as 
low as the devil. 

There are. then, many reasons — many 
needs — forevaluation in continuing edu- 
cation. These needs are personal, pro- 
fessional. and managerial. Since by na- 
ture we already carry on simple habitual 
evaluations all the time, wc can readily 
improve our evaluations through the use 
of more reliable processes. 

Evaluation — what is it? 

In the past few days wc have all been 
following the remarkable achievements 
of Apollo II. The precision with which 
this mission achieved its objectives lias 
been astonishing. What a magnificent 
tribute to the educability of man, that 
three men could on their first attempt 
accomplish with such exactness that 



which no human being has ever before 
accomplished, in an alien, hostile, and 
cruelly restrictive environment. As edu- 
cators we should take heart from this 
magnificent example of the effectiveness 
of teaching and learning for the perform- 
ance of difficult tasks dc novo. Perhaps 
no other human endeavor in all man's 
history has demonstrated so dramatical- 
ly the power and effectiveness of educa- 
tion and training. One recalls Robert 
Theobald’s statement that . . the ta.sk 
of education is to make the impossible 
appear relevant (4).’’ and the hard in- 
sistence of Paul Sharp, president of 
Drake University, that ". . . education 
was created in the firsl place to be use- 
ful (5 t.” Relevant and' useful, indeed. 
In the Apollo moonshots there was a 
vast apparatus to evaluate every aspect 
of tlie mission so that the training, the 
ei|iiipn eat. the flight plan, the activities 
of the astronauts, the feedback of data 
all hor<* immediately on the purposes to 
he achieved. And these data were neces- 
sary sr that appropriate decisions could 
he mm e as required at every step. That 
is what evaluation is: “The provision of 
inform. ition through formal means, such 
as criteria, measurement, and statistics, 
to serve as rational bases for making 
judgments in decision situations (6).” 
Stufflebcam has suggested this ra- 
tionale for scientific evaluation: 

“I. the quality of (educational) pro- 
•rams depends upon the quality 
of decisions in and about the pro- 
grams; 

2. the quality of decisions depends 
upon decision-makers’ abilities to 
identify the alternatives which 
comprise decision situations and 
to make sound judgments of these 
alternatives: 

3. making sound judgments requires 
timely access to valid and reliable 
information pertaining to the al- 
ternatives; 
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4. Mil- availability of such informa- 
lion requires systematic means to 
supply it; and 

5. the processes neecssary for pro- 
viding this information for de- 
cision-making collectively com- 
prise the concept of evalua- 
tion (7)." 

Whether the mission is to land a man 
on the moon or to produce and leach a 
seminar on control procedures in drug 
production, the evaluative process is 
equally accessary' Valid decision mak- 
ing. involving sound judgments among 
alternatives defined hy reliable informa- 
tion, yielded by systematic evaluative 
means — this is our goal in continuing 
education in pharmacy, as well as in all 
other kinds of education. 

We considered earlier that all of us 
habitually engage in casual types of 
evaluation, and in self-clicck probes or 
inquiries to let ns know how we arc 
doing. Do-it-yourself evaluations may be 
extended into more formal evaluative 
studies and. finally, to scientific re- 
search (8). On this scale from the 
simple and informal to the complex and 
formal, the rigor of the evaluation is the 
chief although not the only variable, At 
the upper end (as in the Apollo moon- 
shot) the processes for ensuring exact- 
ness. completeness, and immediacy of 
information relative to making mission- 
achieving decisions is scientific, objec- 
tive. As you go down the scale, informa- 
tion is less reliable, more biased: a “scat 
of the pants” blend of information and 
intuition — like that used by Lindbcrg, 
or the Wright brothers, or Peary, or 
Columbus in their great personal con- 
tributions to the exploration of earth 
space. 

Similarly, many great individually 
conceived continuing education pro- 
grams in pharmacy make landmark 



achievements; yet before us in this rapid- 
ly changing, cvcr-morc-dcmanding world 
are needs and missions which perhaps 
can host he achieved — indeed may only 
he achieved — if a more rigorous process 
of evaluation is employed. 

The rigor of evaluation necessary in 
continuing education may he suggested 
hv pointing out that we must employ a 
process of collecting information, organ- 
izing information, analyzing information, 
and reporting information. Wc need 
criteria for assessing the validity of in- 
formation (is it what the decision-maker 
really needs), the reliability of informa- 
tion (is the information stable, can it he 
replicated again and again), is it timely 
(available when the decision-maker 
needs it), is it pervasive (docs it reach 
all who need it), and is it credible (can 
it he trusted ) (9). 

The evaluation needs of the person 
who designs and provides continuing 
education programs in pharmacy arc not 
unlike those of the Psalmist who cried 
out, 

Lord, make me to know mine 
end. and the measure of my 
days, what it is. that I may 
know how frail I am (10). 

To which as an educator, I can only 
add “Amen.” 

What is the Process of Evaluation? 

If evaluation is to be of long-range 
usefulness to continuing education in 
pharmacy, it must be continuous. Not 
one-shot evaluations but., if possible, an 
evaluation process which is built into 
the very fabric of our programs, a cycle 
of activities which is never ending be- 
cause decision making is never ending, 
because the need of pharmacists to 
learn is never ending. 

A. Preparing goals and objectives. 
The process begins with need discovery. 
The educator in charge of continuing 
education programs in pharmacy is 
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scnsitiu* In needs in pharmacists. in llic 
society served hy pharmacists. These 
si nsit i\ it ics can he sharpened. riTineil. 
validated. ami defined into statements of 
purpose, goals, objectives for programs. 
The objectives must of course be slated 
to include the context, the social and 
learning milieu, in which the objective 
is valid and relevant and useful for the 
learners — the clients who are to profit 
from the program. 

Preparing objectives is a complex ac- 
tivity. Whose objectives? There are a 
number to be considered: Society has 
over-all aims which must be considered, 
but these are usually so broad and so 
abstract that they do not help to define 
specific programs. A profession such as 
pharmacy has its own purposes; but 
these again are likely to he too general 
for mounting specific programs. How- 
ever. social and professional aims must 
certainly be compatible with and rele- 
vant to the objectives of specific con- 
tinuing education programs for pharma- 
cists. 

The teachers of continuing education 
in pharmacy formulate objectives to 
guide their programs. The objectives de- 
scribe the changes the teachers wish to 
bring about; these are action objectives. 
To act. upon them the program developer 
must know, first, the needs and objec- 
tives of his clients. He needs some 
mechanism in addition to sensitivity, 
intuition, and experience to he sure that 
his objectives anil the objectives of his 
clients are reasonably compatible; in 
fact, the teacher-learner objectives must 
derive from a common base, or the pro- 
gram is likely to be ineffectual (II). 

Stating objectives may be viewed as 
a chore which can be by-passed; it is 
easy to follow an action orientation 
which decrees that we get started on the 
job without worrying about writing out 
precisely what we intend to do. Some- 



times this condition prevails simply be- 
cause we do not really know (our in- 
formation system is so faulty) what we 
need to do. and we want to leave open 
clear options as we go along. But note 
that sound decisions cannot he made 
without clear alternatives, valid and re- 
liable information. The expectation that 
we can make better decisions about 
goals and activities later on if wc have 
not first set up a tight information sys- 
tem to develop goals and activities is 
naive, and usually disastrous. 

Learning activities are intended to 
change behavior. Hence objectives 
should he written if possible in behavior- 
al terms. What kind of behavior is de- 
sired as an outcome; in what context 
does the behavior have relevance; what 
content does the learner have to accept 
and employ in achieving the behavior 
change; what persons, agencies, or 
organizations arc the agents in initiating, 
tcaehing, and establishing the new be- 
haviors? 

Rnndchaugh suggests nine criteria for 
shaping educational objectives: 

“I. Are the objectives explicit in 
specifying the area in which the 
changed behavior is to operate? 

2. Arc the objectives definite with 
respect to (lie kind of behavioral 
change to be accomplished? 

3. Arc the objectives stated so as to 
identify those who arc to be in- 
volved? 

4. Arc the objectives the result of 
cooperation between continuing 
education personnel and the oth- 
ers concerned to analyze the 
situation and identify problems? 

5. Arc the objectives compatible 
with the general aims of society 
(the profession of pharmacy, and 
the University)?"' 

* Parentheses indicate slight modification to 
apply more directly to continuing education 
in pharmacy. 
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A. Arc the objectives specific enough 
to serve as a base lor planning, 
conducting and evaluating in an 
action educational program? 

7. Arc the objectives sufficiently 
limited in number to avoid undue 
confusion and diffusion of effort 
on the part of those involved? 

8. Are the objectives achievable, 
considering the level of concern, 
the maturity of persons involved, 
and the resources available? 

9. Are the objectives such that can 
relate intimately to both immedi- 
ate anil long-time educational 
goals, and lead to even higher 
levels of achievement (12)7” 

Objectives which meet the criteria 
above will provide a base for the 
measurement and interpretation of data 
later on. In fact, the absence of ade- 
quate behavioral objectives leaves the 
educator without a valid basis for de- 
termining success or failure of the con- 
tinuing education program. 

li. Knowing your client g roup : se- 
lecting storting points. Of primary im- 
portance in the evaluative process is 
knowing your clients. l*or example, hmv 
can you write behavioral objectives 
unless you know current client (learn- 
er) behaviors well enough to discern 
needed changed behaviors? Knowing 
your clients also implies intimate knowl- 
edge of the total as well as specific en- 
vironment in which your clients live, 
work, and learn. Again, this requirement 
implies an information collection sys- 
tem whose pervasiveness and accuracy 
arc unquestioned. 

I have been fortunate in having the 
opportunity to observe a continuing edu- 
cation program in pharmacy from its 
inception to maturity at the University 
of Wisconsin. Men like Louis Basse, Bill 
Apple. Gus l.embcrger, Dick Strommen, 
and Bill Bloekstein have made it their 
business before anything else to know 



their clients, the total context of phar- 
macy services in the slate, and the needs 
and problems which require tin on-going 
program of continuing education in 
pharmacy. Informal and formal means 
of knowing clients arc required. In this 
process, starting points for continuing 
learning arc discovered, defined, ana- 
lyzed. and processed into learning ob- 
jectives. 

CZontinuing education programs must 
sttirt where the clients— the learners — 
are. This is no empty cliche. If you start 
too high, yon fail; if you start too low, 
you drive clients away in disgust and 
boredom. Of course, the best way to 
establish starting points for learning is 
to have a mechanism for assessing spe- 
cific professional behaviors of pharma- 
cists. With such information as a bench- 
mark. you can then more confidently 
establish continuing learning objectives 
for pharmacists which will be clearly 
recognized as relevant and useful. 

Measurements of behaviors and per- 
formances must be accurate both in the 
initial stages of program conception and 
in the later stages of program evaluation. 
You cannot measure what clients 
learned from your program if you do 
not know the state of their learning be- 
fore the program started. You cannot 
measure achievement of behavioral ob- 
jectives if you do not have at the start 
measures of present behaviors. Standards 
of measure and behavior arc, after all, 
two of the criteria which mark off a pro- 
fession from other human activity. As 
professionals we take pride in our ability 
to measure, to quantify, to analyze, to 
interpret, to conform our behaviors to 
rigorous standards in our fields. I am 
reminded of a delightful passage written 
in the early seventeenth century by a 
famous British lawyer, John Scldon. 
Scldon was writing, as befits a lawyer, 
on equity. He wanted to show how 
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equity differs from law. The passage 
ijocs: . . tor law we have a measure, 

know what to trust; | Inti | equity is ac- 
cording to the conscience ol him that 
is Chancellor. Tis till one as if they 
should make the standard measure we 
call foot a chancellor’s foot; what an 
uncertain measure this would he? One 
Chancellor has a long foot, another a 
short foot, a third an indifferent foot. 
Tis the same in the Chancellor’s con- 
sciencc (13) 

I n a sense, since we are seeking be- 
havioral changes in pharmacists, the 
pharmacists themselves arc part of our 
measurement system. Indeed, if the 
ultimate evaluation of the usefulness of 
pharmacy and pharmacists in society 
is the health and safety of people as a 
result of the work of pharmacists, the 
pharmacist is the starting point, and 
we must measure him in his work with 
people, not from some vague and vacil- 
lating standard of a Chancellor’s con- 
science. but with fixed and agreed upon 
professional measures, such as the fool. 

C. Dctenninin g programs and activi- 
ties. From carefully constructed objec- 
tives based upon a precise knowledge of 
your clients’ behaviors and needs, you 
proceed to determine programs and ac- 
tivities. The plan of work you develop 
includes the specific tasks to be carried 
out, the substantive areas to be em- 
ployed. how you propose to teach your 
clients in their situation, the learning 
activities which you will prepare and 
administer, who will do what and how 
you will arrange for cooperation and 
follow through among personnel, the 
time schedule you must adhere to, how 
you propose to measure your progress 
towards the stated goals, and how you 
propose to make decisions along the 
way if information feedback indicates a 
need for a change in the basic plan. 

Involved in this phase (sometimes 



called "input evaluation") is your ac- 
tivity in organizing and in utilizing re- 
sources to meet program goals and ob- 
jectives, your total instructional strategy 
to solve learning problems, and your 
contingencies for evolving new strategies 
if needed. 

/). Measuring pntgress. When the 
program is placed into operation, the 
pharmacists are signed up. and the 
teaching ami learning begin, the program 
director must be able to tell what is 
happening. Is progress being made to- 
wards objectives? How do you know? 
Know for sure, that is. Here you can 
see the importance of specific behavioral 
objectives at the start, a knowledge of 
clients (possibly brought up-to-date by 
some form of pretest at the start of the 
program) and a detailed plan of work. 
If specific measures do not indicate 
progress towards objectives, do you 
have adequate information for making 
changes in the program? Regular feed- 
back from the learners is needed at this 
stage. As your program moves to com- 
pletion, the data collected should make 
it possible for you to determine not 
only whether objectives arc being 
achieved but also whether new objec- 
tives can be determined for future pro- 
grams which follow up this activity. 
Measuring progress is in effect an evalu- 
ation of your design, for you will un- 
cover here the evidence of good or poor 
planning, good or poor data f,>r de- 
cision making. If your data do not yield 
meaningful results, you arc very much 
in the unenviable position of the poet 
William Wordsworth who in a small 
poem remarked: 

I have often sighed to measure 

by myself a lonely pleasure 

( 14 ). 

If measurement is solely personal and 
intuitive, it is pretty hard to be certain 
of what is happening. 



fil med from best available copy 



American Journal of Pharmaceutical lidiualion 



O 

ERIC 



Interpreting evidence of pr< >gre.\s. 
We need now (o know ;i number of 
things as exactly ns possible, in order 
to analyze and interpret our findings. 

1. Have there been any changes in 
the behavior of learners? What 
changes? Mow do these changes 
relate to the objectives of the pro- 
gram? 

2. Have the learners evidenced other 
changes, altitudinal, substantive, 
or what? 

3. Have the teacher-staff-participants 
undergone any changes? They are 
part of the process, too. and do 
not emerge from programs un- 
touched by their experiences. 

4. Did the planned activities in the 
learning situations perform their 
function? Why or why not? Arc 
changes needed? 

5. What decisions must be made; on 
the basis of what evidence? 

6. What follow-up programming can 
we do from this point? What data 
will we need to start planning a 
follow-up activity and future pro- 
gramming? 

Interpreting data is largely an ex- 
pression of the information-gathering de- 
vices which were built into the teaching 
and learning program. You had to de- 
cide what information was needed to 
give evidence of progress toward goals. 
at what points information was needed, 
the sown' and populations of the in- 
formation required, the instruments and 
techniques which would be used, and the 
plan for organizing and analyzing the in- 
formation. 

F rank Alexander has pointed out that 
educators are prone to assume that evi- 
dence collected regarding continuing 
education programs will be positive 
(15). It is essential therefore that inter- 
preting evidence of progress be objee- 

1 1/0 



live and based upon measurements 
which arc unequivocal. The over-all plan 
should give ample time and emphasis to 
an analysis of .ill the evidence, including 
if necessary follow-up measures after 
the program has formally ended. 

/•'. Using evaluation in continuing edu- 
cation. The purposes and by-products of 
evaluating continuing education have 
been summarized by Alexander (lb). 
Evaluation studies, he points out,' have 
important uses in: 

1. Helping to clarify teaching and 
learning objectives. Struggling 
through an evaluation process not 
only provides knowledge about 
what is happening in teaching and 
learning but also gives all the par- 
ticipants experience and confi- 
dence in the necessary steps in 
defining objectives. 

2. Planning instruction or programs 
on a before testing basis. The 
teacher thus knows the state of 
his learners in terms of behaviors, 
knowledge, readiness for further 
learning, prerequisite skills, errors, 
and inadequacies of background 
and experience. 

3. Motivating learning. If objectives 
have been clarified for learners, 
and are accepted by them, greater 
motivation for learning is gener- 
ated. If objectives are relative and 
useful, intrinsic' motivation is like- 
ly to be elicited, of far greater 
power in learning than the usual 
extrinsic motivations which arc 
characteristics of learners in pro- 
grams whose objectives are less 
relevant and useful. 

4. Providing guidance to learners. 
The information-gathering system 
feeds information back to learners 
as well as teachers so that both can 
be guided towards program objec- 
tives, and the stage set for further 
continuing learning. 
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5. Deveiopment of teachers. leathers 
who carry out cvaliialive programs 
lend to value objective, experi- 
mcnlal. aiul creative approaches lo 
learning and teaching. 

Probably one of the most important 
uses of evaluation in continuing educa- 
tion for pharmacists is in future pro- 
gramming — the continuation, revision, 
discontinuation, or alteration of pro- 
grams (17). Let me repeat the defini- 
tion of evaluation which I used earlier: 
“Hvahiulion is the provision of informa- 
tion through formal means, such as eri- 
teria. measurement and statistics, to 
serve as a rational basis lor making 
judgments in decision situations (18)." 
Looked upon in this professional way, 
evaluation need not threaten continuing 
education workers who may otherwise 
mistakenly fear evaluation as somebody 
else’s effort to find out what they, them- 
selves. are doing. Evaluation is a pro- 
fessional tool for making decisions in 
developing and refining programs. It is 
not a bloody meat axe used against 
teachers to expose personal weaknesses. 

Indeed, properly conceived and ex- 
ecuted. evaluation studies rank as 
scientific research, arc publishable, and 
enhance the reputations of those who 
perform them. 

Why . then, are evaluations of con- 
tinuing education programs in pharmacy 
— and other fields— not more widely and 
regularly used? I suggest the following 
reasons: 

1. Evaluation is still not perceived as 
a necessary part of the professional 
activity of program developers. 

2. Program developers arc generally 
overburdened with program, ad- 
ministrative. personnel, and fiscal 
responsibilities and tend to give 
evaluation a lower priority than 
the bread and butter essentials of 
their offices. 



3. Continuing education programs 
often must operate without sub- 
sidy, or with very minimal subsidy, 
and hence there is frequently no 
dollar support which can be given 
to evaluation processes, which do 
cost time and money. 

4. Program developers often believe 
that they arc not competent to 
mount adequate evaluation studies. 
Rather than to do a job poorly, 
they may decide not to do evalua- 
tions at all. except of the casual 
and informal kind. 

5. Some program developers may 
still feel threatened by the whole 
concept of evaluation, or may even 
see evaluation in some curious way 
as violating the ancient and honor- 
able concept of academic freedom. 

6. A truly professional education cli- 
mate may not exist in the program 
development unit. That is to say. 
there may be a professional cli- 
mate with respect to the substan- 
tive aspects of a pharmacy con- 
tinuing education program but not 
a professional climate with respect 
to the teaching-learning decision- 
making aspects of the program. 

In my paper I hope I have supplied 
the counterarguments to five of the six 
above-listed reasons for not engaging in 
evaluation studies. I have not. however, 
supplied a counterargument for item 
number three, which refers to the pover- 
ty-level financing of many continuing 
education programs. Unfortunately this 
is still a fact, and without an adequate 
budget, adequate evaluation is pretty 
hard to come by. I notice, however, that 
other sessions of this conference have 
addressed themselves to this prob- 
lem. and I hope that continuing educa- 
tion programs in pharmacy will be per- 
sistent in demanding, and successful in 
obtaining, the subsidy or other support 
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necessary to carry on regular evalua- 
tions. 

However, even without adequate sub- 
sidy, you already do carry on casual and 
informal evaluations. One of the best 
ways of drawing in subsidy would be 
grail ually to step up the level of evalua- 
tion to the formal research level. Here 
you can engage the sympathy and con- 
cern of a number of people in a uni- 
versity — the graduate school and the 
school of education among them. Gradu- 
ate students doing theses arc an excel- 
lent and appropriate means of drawing 
in subsidy. 

Suimnary/Conclusion 

We have ranged rather widely in this 
short paper. My fear is that the topic 
has been so broad that it has given lati- 
tude for platitude. Nevertheless, as the 
saying goes, the opus must focus, and 
I have tried to focus on a variety of 
needs — personal, professional, manageri- 
al— for evaluation in continuing educa- 
tion; on what evaluation is — a profes- 
sional tool supplying valid and reliable 
information for you as decision makers 
in pharmacy education; as a process with 
relevance and usefulness in program de- 
velopment, assessment, and the cycle of 
follow through. 

We arc all evaluators. You arc 
evaluating me now as I speak; you arc 
evaluating this conference and one an- 
other. I have tried to suggest that we 
build on this natural and habitual evalua- 
tive behavior, and particularly that we 
use means and methods, instruments, 
and statistics which quantify and quali- 
fy our measures of what is happening in 
our continuing education pharmacy pro- 
grams. Only if we have valid and re- 
liable information about what is hap- 
pening to our learners in our programs 
will we be able to be confident in de- 
veloping, in managing, and in assessing 
our programs. □ 
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